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REMARKS 

Please consider the above amendments and the following remarks. 



The above amendments are made solely to remove multiple dependencies from 
the claims and conform the abstract to U.S. practice. The amendments are fully supported by the 
original specification and no new matter has been added. These amendments are not made in 
view of prior art. 



Applicant believes this application is in condition for allowance. If this belief is 
incorrect, or other issues arise, do not hesitate to contact the undersigned or his associates at the 
telephone number listed below. Please apply any charges not covered, or any credits, to Deposit 
Account No. 50-0591, under Order No. 151 15/171001 from which the undersigned is authorized 
to draw. 



Dated: July 14, 2006 Respectfully submitted, 

.^Jonathan P. Osha 7frc*</*s S*tW*£- 

O Registration No.: 33,986 
OSHA • LIANG LLP 
1221 McKinney St., Suite 2800 
Houston, Texas 77010 
(713) 228-8600 
(713) 228-8778 (Fax) 
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ANNOTATED VERSION SHOWING CHANGES 
IMAGE PROCESSING APPARATUS 

TECHNICAL FIELD 

This invention relates to a technique effectively applicable to the processing of an 
image picked up, or in particular to the processing of an image picked up from a person as an 
object. 

BACKGROUND ART 

In the prior art, a technique is available to correct an image of a person as an object by 
which the skin of the object looks smooth and beautiful. As a specific example of this image 
correction, a technique for locally shading off the edges or the brightness difference of the whole 
image or gradating the area of the skin color component of the whole image by using a e- filter 
(for example, see Patent Document 1). 

Also, as a technique intended to correct or remove the undesired components such as 
wrinkles, spots, rough dry skin or pimples included in a face image, the following technique is 
available (see Patent Document 2). 

First, in respect of a signal value of each pixel forming an image of a face mainly, a 
signal level difference of each pixel with each surrounding pixel is detected by a difference 
detector. This signal level difference is compared with a reference value by a threshold 
determining unit. In accordance with the result of this comparison, an arithmetic unit multiplies 
the signal level difference by a predetermined coefficient, which is added to each pixel value. 
From the result of this addition, the reference value for comparison and the coefficient for the 

1 



22511 

PATENT TRADEMARK OFFICE 



Application No. : 1 0/549,653 Docket No. : 1 5 1 1 5/1 7 1 00 1 

multiplier are selected in accordance with the pixel position and the image. In this way, an 
image free of the undesired components included in the face image is obtained. 

As another conventional technique, a human image processing apparatus is available by 
which a decoration can be added to the body portion of a human image, or especially, the face 
portion or the head portion without any superposition of decorations (see Patent Document 3). 
This human image processing apparatus comprises a means for setting the position and the range 
of a body portion area based on at least one element making up the body portion of a human 
image, and a means for adding a decoration only to the background area other than the body 
portion area. Also, this human image processing apparatus may comprise a means for setting the 
position and the range of the body portion area based on at least one element making up the 
body portion of a human image and a means for adding a decoration along the outer periphery of 
the body portion area. 
[Patent Document 1] 
Patent No. 3319727 
[Patent Document 2] 

Japanese Unexamined Patent Publication No. 2000-105815 
[Patent Document 3] 

Japanese Unexamined Patent Publication No. 2000-022929 

DISCLOSURE OF THE INVENTION 

In the conventional technique, the gradation process is executed for the skin of an 
object aimed at the portion having a color component near to the skin color existing in the 
image. As a result, even the portion having a color component different from the skin color is 
prevented from being gradated and blurred. 
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In the conventional technique, however, any portion of an object or the background 
having a color component near the skin color which may exist in the image is also gradated. 
This poses the problem that even the background other than a person constituting an object, 
which has a color component near to the skin color, is blurred. 

Also, in the conventional technique, the skin color component to be gradated is held 
fixedly. The prior art, therefore, often fails to handle the difference in skin color between 
different races or individuals. In such a case, the gradation process for the skin area cannot 
always be correctly executed. 

This problem of the gradation process is posed also in other image processing as well as 
the gradation process. 

The object of this invention is to solve this problem and prevent the area not to be 
subjected to the image processing such as the gradation process from being subjected to the 
image processing. 

For example, the image processing (such as the gradation process) is carried out only 
for a specified area (the face skin portion, for example) of a person constituting an object for the 
purpose of preventing the background having a color component near to the skin color from 
assuming an unnatural state (blurred, for example) by the image processing. 

Another object of the invention is to carry out the image processing in accordance with 
the difference in skin color between different races or individuals by specifying the color 
component for determining the area to be subjected to the image processing (gradation 
processing, for example) such as the skin color component from the image. 

In order to solve the aforementioned problem, this invention has the following 
configuration. According to a first aspect of the invention, there is provided an image 
processing apparatus comprising a sp e oifio target area specifying means and an image generating 
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means. 

The specific target area specifying means specifies a sp e cific target area determined 
based on the body portion of a person providing an object in the image. The sp e cific target area 
specifying means may be configured so that the user specifies a sp e cific target area manually. 
Specifically, the sp e cific target area specifying means may be so configured that the body 
portion is specified based on the area in the image designated by the user, and with the specified 
body portion as a reference, the s p e cific target area is further specified. For example, the 
sp e cific target area specifying means may be so configured that the area designated by the user is 
specified as a body portion, or the body portion may be specified based on a spot, area, color or 
the shape designated by the user. The sp e cific target area specifying means may be configured 
to specify a sp e cific target area based on the body portion specified in this way. 

Also, the sp e cific target area specifying means may be configured to specify a sp e cific 
target area independently of the input operation of the user. The s p e cific target area specifying 
means, for example, may be configured to include a detection means for detecting the body 
portion of a person providing an object in the image and a specifying means for specifying a 
sp e cific target area based on the body portion detected by the detection means. 

Specifically, the detection means detects the position and the range (size) of the body 
portion of an object. The "body portion" indicates a part or the whole of the body of a person 
including the head, the face, the hands and the torso. The detection means may be configured 
using any existing means. 

The image generating means generates an image by executing the image processing on 
a sp e cific target area specified by the sp e cific target area specifying means. The term "image 
processing" used herein is a process for manipulating the image. An example of image 
processing is such a process as image correction or texture mapping. 

4 



Application No.: 10/549,653 



Docket No.: 15115/171001 



Also, the term "image correction" used herein is a process for manipulating the image 
without changing the property of the object of an image. An example of image correction 
includes the gradation process, edge contrast, brightness correction or color correction. In this 
case, the gradation process is a process for blurring the image portion of wrinkles or spots of the 
skin so that the skin of the person providing an object appears smooth. The gradation process is 
a process executed by using the technique called the smoothing, for example, and carried out by 
removing the high frequency component of the skin image. An example includes the moving 
average filter, the load average filter (including the Gaussian filter) or the 6- filter. 

According to the first aspect of the invention, an image can be acquired which is not 
subjected to the image processing for other than the sp e cific target area specified by the sp e cific 
target area specifying means. Specifically, an image can be acquired in which the image 
processing is carried out only in a sp e cific target area based on the body portion of a person 
making up an object. Therefore, the portion (background, for example) different from the object 
to be subjected to the image processing is prevented from becoming an unnatural state by the 
image processing. In other words, the image processing is prevented from being carried out for 
the portion not intended for by the user. 

In the case where the image processing carried out on the image is the gradation 
process, for example, an image can be acquired in which the portion other than the sp e cific 
target area specified by the sp e cific target area specifying means is not gradated. In other words, 
an image gradated only in a sp e cific target area of a person providing an object can be acquired. 
Thus, the portion (background, for example) other than an object can be prevented from being 
blurred by the gradation process. 

The image generating means may be configured in such a manner as to generate an 
image subjected to the image processing based on the color component of the skin color of a 
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person providing an object, as extracted from the body portion providing a reference of the 
sp e cific target area specified by the sp e cific target area specifying means. 

The above-mentioned configuration of the image generating means makes it possible to 
carry out the image processing corresponding to the skin color of each person providing an 
object. Thus, an image can be acquired in which the image processing is carried out accurately 
for each object of a different skin color in accordance with the difference in skin color between 
different races or different individuals. 

As a configuration having a similar effect, the image generating means may be 
configured to generate an image subjected to the imaging processing on that area within a 
sp e cific target area specified by a sp e cific target area specifying means which has a color 
component equal or similar to the main color component of the body portion providing a 
reference of the sp e cific target area. 

According to a second aspect of the invention, the image generating means of the first 
aspect is configured to have a strength value calculation means, a mask means, an image 
processing means and a color component calculation means. 

The strength value calculation means calculates the strength value indicating the degree 
to which the color component of each pixel is near to a predetermined skin color component for 
each pixel of the image to be processed. The color component can assume a value based on any 
color space, and an example includes the Lab value, the RGB value or the xy value. The skin 
color component is a predetermined value, and stored, for example, in the RAM (random access 
memory) of the image correction unit. The strength value is expressed, for example, by the 
values of 256 grades of 0 to 255. The strength value of 0 indicates, for example, a color 
component farthest from the skin color component, and the strength value of 255 indicates a 
color component nearest (the very skin color component) to the skin color component. 
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In the mask means, the strength value for the pixels in other than a sp e cific target area 
specified by a sp e cific target area specifying means is changed to a value far from the skin color 
component. 

For example, 0 can be employed as a value indicating the farthest color component and 
255 as a value indicating the nearest color component from the skin color component. The mask 
means may generate a mask image to conduct the mask process on an area other than the 
sp e cific target area and multiply the generated mask image by the image indicating the strength 
value of each pixel thereby to set the strength value as described above. In the illustrated case, 
the strength value of the area other than the sp e cific target area is 0 and the strength value in the 
sp e cific target area is not less than 0. 

The image processing means conducts the image processing on an image to be 
processed. The image processing conducted by the image processing means is defined the same 
way as the image processing defined in the explanation of the image generating means. 

The color component calculation means calculates a color component which is nearer to 
the color component of the pixel of the original image as a new color component of each pixel, 
the farther the strength value of the particular pixel from the skin color component on the one 
hand, and calculates a color component which is nearer to the color component of a pixel of the 
image generated by the image processing means as a new color component of the pixel, the 
nearer the strength value of the particular pixel to the skin color component on the other hand. 
The color component calculation means calculates a new color component of each pixel (i.e. the 
color component of each pixel of an image output) based on the strength value calculated by the 
strength value calculation means and the mask means. 

The color component calculation means is configured, for example, in such a manner 
that in the case where the image processing is the gradation process, the larger the strength 
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value, the stronger the color component of the gradated image (the image obtained by gradating 
the original image) is reflected in the image generated. Also, the color component calculation 
means is configured in such a manner that the smaller the strength value, the stronger the color 
component of the original image is reflected in the image generated. By this configuration, the 
effect of the image processing is prevented to extend to an area other than the skin color. In the 
case where this effect is desired most conspicuously, the strength value for the pixels in other 
than a sp e cific target area is set to a value farthest from the skin color component. 

According to a third aspect of the invention, the image generating means according to 
the first aspect of the invention is configured of a strength value calculation means, a mask 
means and an image processing means. According to the third aspect of the invention, the image 
generating means includes no color component calculation means, and the color component of 
an image output is calculated by the image processing means. The strength value calculation 
means and the mask means according to the third aspect are identical with the strength value 
calculation means and the mask means, respectively, of the second aspect. The image 
processing means according to the third aspect, on the other hand, conducts the image 
processing on each pixel of an image to be processed in such a manner that the farther the 
strength value of the pixel from the skin color component, the more the effect of the image 
processing is weakened on the one hand, and the nearer the strength value of each pixel from the 
skin color component, the more the effect of the image processing is strengthened on the other 
hand. This image processing means carries out the image processing based on the strength value 
of each pixel of the image obtained by the strength value calculation means and the mask means. 
In this way, according to the third aspect of the invention, the color component calculation 
means is not required. As a result, the apparatus can be reduced in size, increased in the 
processing speed and reduced in cost. 
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According to a fourth aspect of the invention, the image generating means includes a 
strength value calculation means, an image processing means and a color component calculation 
means. In the fourth aspect, the definition of the strength value calculated by the strength value 
calculation means is different from that in the second aspect. According to the fourth aspect, the 
strength value indicates the degree to which the color component of each pixel is near to the 
main color component representing the body portion providing a reference of a specific target 
area. The strength value calculation means according to the fourth aspect, therefore, calculates 
the strength value indicating the degree to which the color component of each pixel of the image 
to be processed is near to the main color component representing the body portion constituting a 
reference of the sp e cific target area. 

Also, the fourth aspect is different from the second aspect in that the mask means may 
or may not be provided. In the absence of the mask means, the color component calculation 
means naturally calculates a new color component of each pixel without using the strength value 
of a mask image. Further, in the absence of the mask means, the image processing means is 
configured to conduct the image processing on a s p e cific target area of the image to be 
processed. 

Except for the aforementioned three points, the fourth and second aspects have a similar 
configuration. According to the fourth aspect, unlike in the second aspect, the strength value is 
varied with the processing result of the s pecific target area specifying means. Specifically, the 
image processing is conducted in accordance with the skin color of the body portion of each 
person providing an object. In accordance with the difference in the skin color due the race or 
individual differences, therefore, the image processing can be carried out accurately for objects 
of different skin colors. 

According to a fifth aspect of the invention, the image generating means is configured 
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of a strength value calculation means and an image processing means. Also, according to the 
fifth aspect of the invention, like in the third aspect of the invention, the color component 
calculation means is not required, and the color component of the image output is calculated by 
the image processing means. According to the fifth aspect of the invention, therefore, like in the 
third aspect of the invention, the apparatus can be reduced in size and increased in the processing 
speed while at the same time reducing the cost. In the fifth aspect of the invention, like in the 
fourth aspect of the invention, however, the strength value is not determined based on a 
predetermined skin color component, but based on the main color component representing the 
body portion providing a reference of the sp e cific target area. According to the fifth aspect of 
the invention, therefore, the image processing can be carried out accurately for each object of a 
different skin color in accordance with the skin color difference between races or individuals. 
Also, in the fifth aspect of the invention, like in the fourth aspect, the mask means may or may 
not be included. 

In the second to fifth aspects of the invention, the image processing means may be 
configured not to conduct the image processing on the pixels having a strength value in a 
predetermined range. The "strength value in a predetermined range" is defined as a strength 
value indicating an area not desirably constituting an object of the image processing. A specific 
example is a strength value farthest from the skin color component or the main color component 
representing the body portion providing a reference of a sp e cific target area. The color 
component calculation means according to the second or fourth aspect or the image processing 
means according to the third or fifth aspect may be set to generate an output image in such a 
manner that the image processing has no effect on the pixels having a predetermined range of 
strength value. In such a setting, it is not necessary to take the trouble of conducting the image 
processing on the pixels having the predetermined range of the strength value. By omitting the 
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image processing on these pixels, therefore, the time required for the image processing by the 
image processing means can be saved. Especially, in the second and fourth aspects having no 
such configuration, the image processing means conducts the image processing regardless of the 
strength value of each pixel. Then, the process corresponding to the strength value is executed 
by the color component calculation means, with the result that the image processing conducted 
by the image processing means may not be reflected in some pixels at all. The configuration in 
which the image processing means determines whether the image processing is to be conducted 
or not in accordance with the strength value, therefore, is effective especially in the second and 
fourth aspects. 

According to a sixth aspect of the invention, the image generating means determines the 
contents of the image processing to be conducted, based on the size of the body portion 
providing a reference of a sp e cific target area specified by the predetermine area specifying 
means. The image generating means generates, for example, the parameters for carrying out a 
predetermined image processing, based on the size of the body portion providing a reference of a 
sp e cific target area. Examples of the parameters include the degree of gradation (more 
specifically, the size of the radius of the Gaussian filter in the case where the gradation process 
is executed using the Gaussian filter) of the gradation process, the degree of edge contrast and 
the degree of brightness correction. Examples of determining the type of image processing 
includes the determination as to whether or not the edge contrast is taken or not, whether the 
brightness correction is carried out or not, and whether the gradation process is carried out or 
not. 

The gradation process is explained as an example. In the case where the gradation 
process is executed excessively for a small area, the whole area would be blurred and the desired 
image (the image corrected to secure an appropriately smooth skin, for example) could not be 
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obtained. In the case where an excessively small gradation process is executed for a large area, 
on the other hand, the points to be gradated (for example, the undesirable portions included in 
the face image such as wrinkles, spots, rough dry skin, pimples, etc.) could not be sufficiently 
blurred and the desired image could not be obtained. This is also the case with other image 
processing such as the degree of edge contrast. In view of this problem, according to the sixth 
aspect of the invention, the proper contents of the image processing are determined and carried 
out in accordance with the size of the body portion providing a reference of the sp e cific target 
area to obtain the image desired by the user. 

Also, the image generating means may be configured to determine the contents of the 
image processing to be conducted, based on the size of the body portion detected by a detection 
means. 

A seventh aspect of the invention further comprises an element extraction means. The 
element extraction means extracts at least an element making up the body portion of a person 
providing an object in an image to be processed, which element is included in a sp e cific target 
area. The "element" indicates a part making up the body portion. Examples of the element 
include parts of the face (specifically, eyes, eyelids, lips, nose, nostrils, eyebrows, eyelashes, 
etc.). In this case, the face constitutes a body portion and the face parts the elements. The 
element extraction means may employ any existing techniques. 

In the seventh aspect of the invention, the image generating means conducts the image 
processing, in limited way, on an element area determined based on the element extracted by the 
element extraction means. Specifically, the image generating means may be configured not to 
conduct the image processing in the specific target area described above. Alternatively, the 
image generating means may be configured to conduct the image processing on the sp e cific 
target area with different parameters (by suppressing the degree of image processing) as 
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compared with the other areas where the image processing is conducted. 

According to the seventh aspect of the invention, the image generating means carries 
out the limited image processing on an element area determined based on the element extracted 
by the element extraction means. As a result, the effect of the image processing on the element 
area is suppressed. According to other aspects of the invention, the contents of the image 
processing are determined based on the body portion providing a reference of the sp e cific target 
area and the color component. In this case, with regard to the pixels having a color component 
near to the skin color component in a s p e cific target area or the main color component 
representing the body portion providing a reference of the sp e cific target area, the elements 
corresponding to the elements described above are subjected to the image processing 
unconditionally. Actually, however, the desire exists to limit (suppress) the effect of the image 
processing on these elements. Depending on the method of makeup, for example, the color of 
the lips (color of the lipstick) or the elements such as the eyelashes may be configured of a color 
component near to the skin color component. In such a case, it may not be desired to conduct 
the same image processing (such as gradation) on the elements as on the other skin portion. The 
seventh aspect of the invention effectively applies in such a case. Specifically, the image 
processing on the elements not sufficiently distinguishable by the strength value calculation 
means or the mask means can be accurately suppressed. 

In the second to fifth aspects of the invention, the image generating means may be 
configured to further include an edge mask means. The edge mask means acquires the edge 
strength of each pixel of the image to be processed, and produces a strength value indicating the 
fact that the higher the edge strength extracted, the farther the skin color component or the main 
color component representing the body portion providing a reference of the sp e cific target area. 
By acquiring the strength value based on the edge strength, the edge of the element making up 
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the body portion of an object can be acquired as a pixel having a strength value indicating that 
the color component is far. The edge mask means may be configured to produce the same edge 
strength for the pixels located within a predetermined range from a given pixel as for the 
particular given pixel. 

In the second to fifth aspects of the invention having the above-mentioned 
configuration, the color component calculation means and the image processing means are 
configured to further acquire a new color component for each pixel based on the strength value 
acquired by the edge mask means. With the second to fifth aspects of the invention having this 
configuration, a similar effect to the seventh aspect can be produced. 

Also, in the second to fifth aspects of the invention having this configuration, the edge 
mask means may be configured to enlarge, after reducing, the image to be processed to the 
original size by applying a strength value to each pixel. 

The edge extracted by the edge mask means is subjected to the limited image 
processing described above. Take the process of gradating the skin component not required as 
an example. If the unrequired skin component is detected as an edge by the edge mask means, 
the gradation process for this unrequired skin component would fail to work effectively. It is 
necessary, therefore, to control the edge mask means not to extract the edge of the unrequired 
component. 

An image being reduced loses the minuscule edge information of the original image. In 
the case where the edge strength is acquired for a reduced image, therefore, the minuscule edge 
of the original image cannot be acquired. The unrequired skin component to be gradated is often 
configured of a minuscule edge. Taking advantage of this feature, the aforementioned 
configuration is employed, so that the edge acquisition for the unrequired skin component by the 
edge mask means can be prevented. Specifically, the above-mentioned configuration of the edge 
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mask means makes possible the desired gradation process. A similar effect can be produced also 
by executing the edge extraction after the smoothing process using a median filter or by setting a 
large radius of the filter used for the edge extraction process. Since the edge is extracted in the 
reduced image, on the other hand, the time required for edge extraction can be reduced. 

According to an eighth aspect of the invention, there is provided an image processing 
apparatus comprising an image specifying means and an image processing means. The image 
specifying means specifies the position and the range of an area including an arbitrary image in 
the image. The arbitrary image is defined as an image to be processed by the image processing 
means, and may assume any form. For example, it is a part or the whole of the human body 
such as the face or the hands or such an object as a food or an automobile or the background 
such as the sky or a mountain. 

The image specifying means may be configured using any existing technique. The 
image specifying means can be configured, for example, to carry out the pattern matching using 
a pattern having a similar shape to an arbitrary image involved. 

The image processing means generates an image subjected to the image processing in 
an area which is located in an area specified by the image specifying means and which has a 
color component equal or similar to the main color component representing the particular 
specified area. An example of the image processing executed by the image processing means is 
the one using a low-pass filter or a high-pass filter. There are also other various processes such 
as the image processing defined in the first aspect, the color inversion process and the image 
rotation process. 

According to the eighth aspect of the invention, even in an area within the specified 
area, the execution of the image processing can be prevented for the portions different from the 
main portion. In the case where it is desired to change the color of only the vehicle body (main 
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portion), for example, the color change of the window glass and the bumper (portions different 
from the main portion) can be prevented. 

The first to eighth aspects of the invention may be implemented also by executing a 
program on an information processing system. Specifically, according to this invention, the 
process executed by each means in the first to eighth aspects described above can be specified as 
a program to be executed by the information processing system or a recording medium for 
recording the particular program. 

According to this invention, the image correction can be carried out only in a specified 
limited area of a person making up an object. Therefore, the portions (such as the background) 
different from the object are prevented from assuming an unnatural state as the result of image 
correction. Also, the image processing can be carried out in accordance with the skin color 
difference from one race or individual to another. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a diagram showing the function blocks of an image correction unit according to 
a first embodiment; 

Fig. 2 is a diagram showing the processing flow of the image correction unit according 
to the first embodiment; 

Fig. 3 is a diagram showing an outline of the mask process; 

Fig. 4 is a diagram showing an outline of the skin color area extraction process 
according to the first embodiment; 

Fig. 5 is a diagram showing an outline of the skin color strength extraction process; 
Fig. 6 is a diagram showing an example of the histogram of the skin color component; 
Fig. 7 is a diagram showing an outline of the skin beautifying process according to the 
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first embodiment; 

Fig. 8 is a diagram showing an example of the n x n operator; 

Fig. 9 is a diagram showing the function blocks of the image correction unit according 
to a second embodiment; 

Fig. 10 is a diagram showing the processing flow of the image correction unit according 
to the second embodiment; 

Fig. 1 1 is a diagram showing function blocks of the image correction unit according to a 
third embodiment; 

Fig. 12 is a diagram showing the processing flow of the image correction unit according 
to the third embodiment; 

Fig. 13 is a diagram showing an example of the Sobel filter; and 

Fig. 14 is a diagram showing an example of the difference between the skin color 
strength image and the edge mask image. 

BEST MODE FOR CARRYING OUT THE INVENTION 

[First embodiment] 

Next, the image correction unit according to an embodiment of the invention is 
explained with reference to the drawings. The description that follows deals with an image 
correction unit la as a specific example of the image correction unit according to a first 
embodiment to correct the image of the skin area in a human image. Specifically, the gradation 
process is explained as an example of the image processing to be carried out. The image 
correction unit la may be used also for the image other than the human image such as a vehicle 
or a landscape. In this case, various image processing are conceivable including the image 
correction for changing the color or the image correction using a high-pass filter. 
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In this description, the human image is an image including at least a part or the whole of 
the human face. The human image, therefore, may include the image of the person as a whole or 
the image of only the face or the upper half portion of the person. Also, an image of a plurality 
of persons may be included. Further, any pattern of a landscape (background: including an 
object watched as an object) or a figure other than a person may be included in the background. 

The explanation of this embodiment is illustrative and the configuration of the invention 
is not limited to the following description. 
[System configuration] 

The image correction unit la, in terms of hardware, is configured of a CPU (central 
processing unit), a main storage (RAM) and an auxiliary storage unit connected through a bus. 
The auxiliary storage unit is configured of a nonvolatile storage unit. The term "nonvolatile 
storage unit" as used herein designates what is called the ROM (read-only memory) (including 
the EPROM (erasable programmable read-only memory), EEPROM (electrically erasable 
programmable read-only memory and the mask ROM), FRAM (ferroelectric RAM) and the hard 
disk. 

Fig. 1 is a diagram showing the function blocks of the image correction unit la. The 
image correction unit la, in which the various programs (OS, applications, etc.) stored in the 
auxiliary storage unit are loaded in the main storage unit and executed by the CPU, functions as 
a device including a face detector 2, a mask processor 3, a skin color area extractor 4a, a skin 
beautifying processor 5a and a storage St. The face detector 2, the mask processor 3, the skin 
color area extractor 4a and the skin beautifying processor 5a are implemented by the CPU 
executing the image correction program according to the invention. 

The storage St is configured of what is called the RAM. The storage St is used for the 
various processes executed by the face detector 2, the mask processor 3, the skin color area 
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extractor 4a and the skin beautifying processor 5a. The data read from or written into the 
storage St, for example, include the data of an original image 6 to be processed, a mask image 7 
as an intermediate data, a skin color strength image 8, a masked skin color strength image 9, a 
skin color area image 10a and a gradation image 1 1 or the data of the skin beautification image 
12 as an output data. 

Fig. 2 is a diagram showing the process executed by each functional unit shown in Fig. 
1 and the overall processing flow of the image correction unit la. With reference to Fig. 13, 
each functional unit is explained below. 
<Face detector> 

The face detector 2 executes the face detection process. The face detection process is 
explained below. In the face detection process, the data of the original image 6 is input and the 
face position detection step SOI is executed thereby to output a face rectangle coordinate. 
Specifically, in the face detection process, a face is detected as a body portion of an object. 
From this face rectangle coordinate, the position of the face of the person constituting an object 
in the original image 6 is specified. 

The data of the original image 6 is the data of the human image input to the image 
correction unit la. The face rectangle is recognized as a rectangle including the face portion of a 
person contained in the original image 6 (hereinafter referred to as the face rectangle. See 
reference numeral 17 of Fig. 3(a)). The face rectangle coordinate is the data indicating the 
position and size of the face rectangle in the original image 6. 

The face position detection step SOI may be implemented by any existing method (for 
example, see Patent Document 3). The face rectangle coordinate can be obtained, for example, 
by template matching using a reference template corresponding to the contour of the whole face. 
Also, the face rectangle coordinate can be obtained by template matching based on the face 
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component elements (eyes, nose, ears, etc.). Further, the top of the hair may be detected by the 
chroma key process, and based on this top, the face rectangle coordinate can be obtained. Also, 
at the face position detection step SOI, the face rectangle or the face rectangle coordinate may be 
specified manually by the user. In similar fashion, the face rectangle or the face rectangle 
coordinate can be specified based on the information input by the user, i.e. semiautomatically. 
<Mask processor> 

The mask processor 3 executes the mask process. The mask process is explained 
below. In the mask process, the face rectangle coordinate is input and the mask image 
generation step S02 is executed thereby to output the data of the mask image 7. 

In the mask image generation step S02, the face and the area under the face of a person 
constituting an object are estimated based on the position of the face of the person constituting 
the object, i.e. the face rectangle coordinate input in this apparatus 1 thereby to generate a mask 
image to mask other than the estimated area. In other words, at the mask image generation step 
S02, a sp e cific target area (the face and the area under the face in the case under consideration) 
is specified based on the position of the face as a body portion, and the mask image 7 is 
generated to mask other areas. In this way, according to this embodiment, the face detector 2 
and the mask processor 3 are used as an example of a sp e cific target area specifying means. 
Also, according to this embodiment, the mask processor 3 is used as an example of the 
specifying means and the mask means. 

Fig. 3 is a diagram showing an outline of the mask image generation step S02. In the 
mask image generation step S02, first, the coordinates of two ellipses 13, 14 corresponding to 
the input face rectangle coordinate are calculated using the equation of Expression 1 shown 
below. Specifically, first, the width (w) and the height (h) of the face rectangle are calculated or 
input. By multiplying the preset elliptic ordinate coefficient (pO, pi) and the elliptic abscissa 
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coefficient (qO, ql) by w and h, respectively, the length of the long and short axes of the two 

ellipses (aO, bO, al, bl) can be obtained. 

The ellipse 13 is a figure indicating the area of the human face providing an object and 

the ellipse 14 a figure indicating the area of the parts (neck, chest, shoulder, etc.) under the face 

of the person providing the object. According to this invention, the ellipse 13 is set in contact 

with four points of the face rectangle 17. On the other hand, the ellipse 14 is set to circumscribe 

on the lowest portion of the ellipse 13 with the long axis thereof kept horizontal. 

[Expression 1] 

aO = h x pO 
b0~w xqO 
al = h xpl 
bl « w xql 

Next, at the mask image generation step SOI, the two ellipses 13, 14 obtained are 
enlarged to obtain ellipses 15, 16, respectively. The ellipses 13, 15 have the same center (the 
intersection of the long and short axes) and so do the ellipses 14, 16. Using the ellipses 13 to 16 
thus obtained, the mask image 7 is acquired. 

For example, the inside of the ellipse 13 and the inside of the ellipse 14 are set as a 
transmission area (the area not masked). Next, in the area between the ellipses 15, 13 and the 
area between the ellipses 16, 14, the transmittance gradation is created with the transmission 
ratio increasing from outside (ellipses 15, 16 side) toward inside (the ellipses 13, 14 side). This 
gradation may be either linear or nonlinear. The area constituting the outside of the ellipse 15 
and the outside of the ellipse 16 is set as a non-transmission area (masked area). 

By this mask image generation step S02, the data of the mask image 7 is output. The 
mask image 7 may be generated using any figure other than the ellipse. For example, it may be 
generated using a special figure in the shape of the upper half portion of the person. 
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<Skin color area extractor> 

The skin color area extractor 4a executes the process of extracting the skin color area. 
The skin color area extraction process is explained below. In the skin color area extraction 
process, the data of the original image 6, the face rectangle coordinate and the data of the mask 
image 7 are input, while the skin color strength extraction step S03, the synthesis step S04a and 
the skin color area correction step SOS are executed thereby to output of the data of the skin 
color area image 10a. In this way, according to this embodiment, the skin color area extractor 4a 
is used as an example of the strength value calculation means. 
«Skin color strength extraction process» 

Fig. 4 is a diagram showing an outline of the skin color area extraction process. In the 
skin color area extraction process, first, the data of the original image 6 and the face rectangle 
coordinate are input and the skin color strength extraction step S03 is executed. 

Fig. 5 is a diagram showing an outline of the skin color strength extraction step S03. 
Fig. 6 is a diagram showing a histogram of the skin color component used in the skin color 
strength extraction step S03. The skin color strength extraction step S03 is explained below with 
reference to Figs. 5, 6. 

At the skin color strength extraction step S03, first, the sampling area 18 is specified 
inside the face rectangle 17 using the input face rectangle coordinate. The sampling area 18 is 
specified by, for example, the center coordinate of the face rectangle 17 and the values w and h 
of the face rectangle 17 multiplied by a constant. The sampling area 18 may alternatively be 
specified by other methods. The sampling area 18 is desirably set in such a manner as not to 
include areas such as the eyes or nostrils apparently having a different color from the skin. 

Next, at the skin color strength extraction step S03, the pixel value (color component 
value) in the sampling area 18 is sampled (skin color sampling). In this sampling process, the 
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skin color of the face of the object is mainly sampled. Based on the value of the sampled color 
component, the histogram shown in Fig. 6 is formed. Fig. 6 shows an example of the histogram 
formed based on the Lab color space. Once the histogram is formed, the components 10 % 
(hatched in Fig. 6) above and below the abscissa (value of L or a, b) is cut off. The numerical 
value of 10 % referred to herein can be appropriately changed by the designer. After that, using 
the Lab value for the portion not cut in the skin color component histogram, the standard 
deviation and the center of gravity in the sampling area 18 are calculated. In accordance with 
the equation of Expression 2 using the six values thus calculated, the degree of the skin color 
(hereinafter referred to as the skin color strength, which corresponds to the strength value) of 
each pixel of the original image 6 is calculated (the skin color strength extracted) thereby to 
generate the skin color strength image 8. 
[Expression 2] 



where L\ a', b' are the center of gravity of the Lab value in the sampling area, and W L , W a , Wb 
the standard deviation of the Lab value of the sampling area multiplied by a constant. 

In forming the skin color component histogram, the accumulated portion representing 
10 % from the ends of the abscissa in Fig. 6 are cut off and therefore the noise component is 
removed so that the distribution of the skin color component can be acquired more accurately. 
The term "noise component" referred to herein indicates the information on the pixels mainly 
having the color component other than the skin color such as the nostrils and eyes in the 
sampling area 18. This process makes it possible to delete the information on the color 
components other than the skin color such as the nostrils and eyes, if included in the sampling 
area 18. 
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«Synthesis process» 

Next, in the skin color area extraction process, the data of the skin color strength image 
8 and the data of the mask image 7 are input and the synthesis step S04a is executed. 

At the synthesis step S04a, the input skin color strength image 8 and the mask image 7 
are synthesized with each other. Specifically, the skin color strength image 8 generated by the 
skin color strength extraction step S03 and the mask image 7 generated by the mask process are 
multiplied by each other. The execution of the synthesis step S04a generates the masked skin 
color strength image 9. 
«Skin color area correction process» 

Next, in the skin color area extraction process, the data of the masked color strength 
image 9 is input and the skin color area correction step SOS is executed. 

At the skin color correction step S05, the reduction process is executed on the masked 
skin color strength image 9 generated at the synthesis step S04a. The execution of the reduction 
step reduces the skin color strength around the eyes and mouth. Specifically, the black area 
(area of low or zero skin color strength) not to be gradated is expanded outward. This reduction 
step prevents the gradation process from being executed for the parts surrounding the eyes and 
mouth. In other words, the parts around the eyes and the mouth of which a clear image is to be 
acquired are prevented from being blurred. The execution of the skin color area correction step 
S05 generates the skin color area image 10a. In the skin color area image 10a, the pixels having 
a high skin color strength are expressed with a large pixel value, and vice versa. 
<Skin beautifying processor> 

The skin beautifying processor 5a executes the skin beautifying process. The skin 
beautifying process executed by the skin beautifying processor 5a is explained below. In this 
skin beautifying process, the data of the original image 6 and the data of the skin color area 
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image 10a are input, and the gradation filter step S06a and the skin beautifying synthesis step 
S07 are executed thereby to output the data of the skin beautification image 12. In this way, 
according to this embodiment, the mask processor 3, the skin color area extractor 4a and the skin 
beautifying processor 5a are used as an example of the image generating means. Also, 
according to this embodiment, the skin beautifying processor 5a is used as an example of the 
color component calculation means. This data of the skin beautification image 12 is output by 
the image correction unit la. 
«Gradation filter process» 

First in the skin beautifying process, the data of the original image 6 and the data of the 
skin color area image 10a are input and the gradation filter step S06a executed. At the gradation 
filter step S06a, the original image 6 is gradated. The term "gradation" used herein may be any 
existing gradation process. A method using the moving average filter, the load average filter 
(including the Gaussian filter) or the 8- filter is an example. 

At the gradation filter step S06a, only the pixels of the original image 6 which have the 
skin color strength value larger than 0 in the skin color area image 10a are gradated. As a result, 
the pixels 0 in the color strength value, i.e. the pixels which apparently represent no skin color or 
the pixels masked by the synthesis step S04a are not gradated. The execution of the gradation 
filter step S06a generates a gradated image 1 1 . 
«Skin beautifying synthesis process» 

Next, in the skin beautifying process, the data of the original image 6, the data of the 
skin color area image 10a and the data of the gradated image 11 are input and the skin 
beautifying synthesis step S07 is executed. In the skin beautifying synthesis step S07, the 
original image 6 and the gradated image 1 1 are synthesized translucently using the skin color 
strength of the skin color area image 10a. Expression 3 represents equations for the translucent 
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synthesis executed at the skin beautifying synthesis step S07. 
[Expression 3] 

G = G org x(l-V) + G smooth xV 

where Ro rg , G org , B org are RGB components of the original image, Rs m0 oth, G sm ooth, B smoo th the 
RGB components of the gradated image, and V the skin color strength (0 to 1) of the skin color 
area image. 

In the translucent synthesis using Expression 3, the synthesis process is executed in 
accordance with the skin color strength. Specifically, the pixels having a high skin color 
strength strongly reflect the pixel value (RGB components) of the gradated image 11, while the 
pixels having a low skin color strength strongly reflect the pixel value (RGB components) of the 
original image 6. By this translucent synthesis, the gradation degree is strengthened for the area 
having a high skin color strength (i.e. the skin color area) while at the same time weakening the 
gradation degree for the area having a low skin color strength (i.e. the area other than the skin 
color area). The execution of the skin beautifying synthesis step S07 generates the skin 
beautification image 12. 
[Operation/effects] 

In the image correction unit la according to this invention, the face of an object is 
detected from the image to be processed and the face rectangle coordinate obtained at the face 
position detection step SOI. Based on the face rectangle coordinate, the mask image 7 is 
generated to mask other than the upper half portion of the object. In the skin beautifying 
process, the gradation process reflecting the mask process using the mask image 7 is executed. 
As a result, during the execution of the gradation process in the area having the skin color 
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component such as the face of the object, the gradation process is not executed for the area (such 
as the background) other than the object having the skin color component in the same image. 
While the gradation process is being executed on the face of the object, etc., therefore, the 
background, etc. having the skin color component is prevented from being blurred and can be 
maintained in a clear state. Specifically, only the face of the object or thereabouts are smoothed 
by removing wrinkles and spots. 

Also, in the image correction unit la according to this invention, the skin color 
component of the object is extracted from inside the sampling area 18, i.e. from inside the face 
area of the object detected by the face position detection step SOI. Based on the skin color 
component thus extracted, the area to be gradated is determined. Specifically, based on the skin 
color component extracted, the skin color component recognized as a skin color at the time of 
generating the skin color strength image 8 is determined. In the case where the object is a 
person having a white skin, therefore, the skin color strength image 8 is generated based on the 
extracted white skin color component, while in the case where the object is a person having a 
black skin, on the other hand, the skin color strength image 8 is generated based on the extracted 
black skin color component. In this way, in the image correction unit la according to this 
invention, the skin color is not determined fixedly, but sampled from the face position in the 
original image 6. Therefore, it is possible to handle the difference in skin color between 
different races or individuals, thereby producing a stable correction effect. 

Also, at the mask image generation step S02, the gradation of non-transmissibility is 
applied between the outer ellipse and the inner ellipse. In the boundary between the 
transmissible area not masked and the non-transmissible area masked, therefore, an unnatural 
image is prevented from being generated. 

By the way, the image correction unit la according to this invention may be mounted 
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on any of the various existing devices such as a printer, a display, a digital camera or a MPEG 
(moving picture experts group) player. In such a case, the image data input to the devices are 
input to the storage St as the data of the original image 6. The data of the skin beautification 
image 12 output from the image correction unit la is used in accordance with the characteristics 
of each device. In the case where the image correction unit la is mounted on the printer, for 
example, the skin beautification image 12 is printed on the printer. 

Also, the image correction unit la according to this invention can be virtually 
implemented on the information processing system having a CPU by executing the steps SOI to 
S07 in Fig. 2. In this case, the program for causing the information processing system to execute 
the steps SOI to S07 constitutes this invention. This program is recorded in a recording medium 
such as a CD-ROM and may be executed directly by a personal computer or a server (the server 
installed on an ASP (application service provider), for example), or stored in and executed by a 
nonvolatile storage unit such as a hard disk drive or a ROM. In this case, the data of the original 
image 6 may be input from a scanner or a digital camera connected to the information 
processing system. Also, the data of the original image 6 may be input by uploading or 
downloading from other devices through a network such as the internet. 
[Modifications] 

The face detector 2, the mask processor 3, the skin color area extractor 4a and the skin 
beautifying processor 5a each may be configured of a chip mounted as hardware. Also, the 
storage St may be configured of a RAM of another device on which the image correction unit la 
is mounted. Specifically, the storage St is not necessarily arranged in the image correction unit 
la, but outside of the image correction unit la as far as it is accessible from the face detector 2, 
the mask processor 3, the skin color area extractor 4a and the skin beautifying processor 5a. In 
this case, the storage St may be configured to be shared by other devices (such as the CPU of a 
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device on which the image correction unit la is mounted) and each of the processors 2 to 5 of 
the image correction unit la. 

Also, at the gradation filter step S06a, the gradation degree can be determined based on 
the size of the face rectangle detected by the face position detection step SOI. Specifically, the 
larger the face rectangle, the stronger (larger) the gradation degree with which the gradation 
process is executed. On the contrary, the smaller the face rectangle, the weaker (smaller) the 
gradation degree with which the gradation process is executed. This can be realized, for 
example, by manipulating the parameters such as the radius of the moving average filter or the 
load average filter. In the case of the Gaussian filter, on the other hand, the gradation degree can 
be changed by changing the standard deviation a in the equation below. 
[Expression 4] 



In the case of a simple smoothing filter, the gradation degree can be changed by 
changing the value n in the n x n operator. Fig. 8 is a diagram showing a specific example of the 
n x n operator. Fig. 8(a) shows an example of the operator in a case where n = 3, Fig. 8(b) a 
case where n = 5, and Fig. 8(c) a case where n = 7. The larger the value n, the larger the 
gradation degree. 

This configuration makes possible the execution of the gradation process with the 
gradation degree suitable for the face size. It is thus possible to obviate the problem of the 
blurring of the whole face which otherwise might be caused by executing the gradation process 
with an excessively large gradation degree for a small face. 

Also, in the skin beautifying process, the image processing (examples: edge contrast, 
brightness correction, color correction, texture mapping) other than the gradation process as well 
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as the gradation process may be executed. 

Further, the mask processor 3 is not necessarily provided. In the absence of the mask 
processor 3, however, the process based on the mask image 7 is not executed. As a result, the 
time required before acquiring the skin beautification image 12 may be lengthened. 
[Second embodiment] 
[System configuration] 

Next, an image correction unit lb according to a second embodiment is explained. Fig. 
9 is a diagram showing the function blocks of the image correction unit lb. The image 
correction unit lb is different from the image correction unit la in the provision of a skin 
beautifying processor 5b in place of the skin beautifying processor 5a. The points of the image 
correction unit lb different from the image correction unit la are explained below. 

Fig. 10 is a diagram showing the process executed by each functional unit shown in Fig. 
9 and the overall processing flow of the image correction unit lb. With reference to Figs. 9 and 
10, each functional unit of the image correction unit lb is explained below. 
<Skin beautifying processor> 

The skin beautifying processor 5b is different from the skin beautifying processor 5a in 
that the skin beautifying synthesis step S07 is not executed and the gradation filter step S06b is 
executed in place of the gradation filter step S06a. The skin beautifying process executed by the 
skin beautifying processor 5b is explained below. 
«Gradation filter process» 

In the skin beautifying process executed by the skin beautifying processor 5b, the data 
of the original image 6 and the data of the skin color area image 10a are input and the gradation 
filter step S06b is executed. At the gradation filter step S06b, each pixel of the original image 6 
is subjected to the gradation process corresponding to the skin color strength included in the skin 
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color area image 10a. Specifically, the gradation process for the pixels having a high skin color 
strength is set to a large gradation degree, and the gradation process for the pixels having a low 
skin color strength to a low gradation degree. Also, the gradation filter step 06b may be 
configured in the manner described below. 

In the image correction unit la, the gradated image 11 is generated by the gradation 
filter step S06a, and the skin beautification image 12 is generated by the skin beautifying 
synthesis step S07 using the gradated image 11, the original image 6 and the skin color area 
image 10a. In the skin beautifying correction unit lb, on the other hand, the gradated image 1 1 
may not be generated but the skin beautification image 12. Specifically, each time the value of 
each pixel of the skin beautification image 12 is calculated based on the equation of Expression 
3, the gradation process is executed for the pixels to be processed. Specifically, each value of 
the RGB components of the gradated image used in Expression 3 is calculated each time only 
for the required pixels. This configuration eliminates the need of buffering the gradated image 
1 1 and can save the memory area. 
[Operation/effects] 

In the image correction unit lb according to this invention, the skin beautifying process 
is executed in such a manner that the skin beatification image 12 is generated directly as an 
output image without generating the gradated image 11. Therefore, the time required for the 
gradation filter step S06a to generate the gradated image 11 and the skin beautifying synthesis 
step S07 can be reduced. 
[Third embodiment] 
[System configuration] 

Next, an image correction unit lc according to a third embodiment is explained. Fig. 1 1 
is a diagram showing the function blocks of the image correction unit lc. The image correction 
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unit 1c is different from the image correction unit lb in the provision of the skin color area 
extractor 4c in place of the skin color area extractor 4a and the provision of the edge mask 
processor 19. The points of the image correction unit lc different from the image correction unit 
lb are described below. 

Fig. 12 is a diagram showing the process executed by the functional units shown in Fig. 
1 1 and the overall processing flow of the image correction unit lc. With reference to Figs. 1 1 
and 1 1, each functional unit of the image correction unit lc is explained. 
<Edge mask processor> 

The edge mask processor 19 executes the edge mask process. The edge mask process is 
explained below. In the edge mask process, the original image 6 is input, and the edge mask 
image generating step S08 is executed to output the data of the edge mask image 20. 

First, in the edge mask image generating step S08, the input original image 6 is reduced 
and a reduced image is acquired. By inputting the size of the face rectangle further, for example, 
the reduction ratio can be determined based on the size of the face rectangle. For example, the 
width of the largest one of the input face rectangles can be reduced to about a predefined number 
of pixels (about several tens to one hundred pixels). 

Next, the edges are extracted based on the reduced image, i.e. the edge strength is 
acquired. This edge extraction process can be executed by any existing technique. The edge 
extraction using the Sobel filter, for example, is carried out. Fig. 13 is a diagram showing an 
example of a Sobel filter. Fig. 13(a) shows a downward Sobel filter and Fig. 13(b) an upward 
Sobel filter. The edge extraction process using each Sobel filter is executed, and the edge image 
of each Sobel filter is acquired. In this case, two edge images are acquired. 

Next, each acquired edge image is grayed and synthesized thereby to acquire a 
synthesized edge image. By this synthesis process, the edge extracted by the downward Sobel 
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filter and the edge extracted by the upward Sobel filter are indicated in the synthesized edge 
image. 

Next, the acquired synthesized edge image is inverted, and an inverted edge image is 
acquired. Next, the inverted edge image is subjected to the reduction process. By the execution 
of the reduction process, an image with the extracted edge expanded in the surrounding area is 
acquired. Then, the inverted edge image subjected to the reduction process is enlarged to the 
size of the original image 6 thereby to acquire the edge mask image 20. In the subsequent 
processes, the pixel value in the edge mask image 20 is handled as a skin color strength. 
Specifically, the pixels of the edge portion acquired have a low or zero pixel value due to the 
inversion and therefore are handled as pixels having a low skin color strength. Also, due to the 
reduction process, the effect of the extracted edge extends to the surrounding area. Specifically, 
the edge mask image 20 is generated as an image indicating the low skin color strength of the 
extracted edges and the surrounding area. 
<Skin color area extractor> 

The skin color area extractor 4c is different from the skin color area extractor 4b in that 
the synthesis step S04c is executed in place of the synthesis step S04a. The skin color area 
extraction process executed by the skin color area extractor 4c is explained below with reference 
to the synthesis step S04c in particular. 
«Synthesis process» 

In the skin color area extraction process executed by the skin color area extractor 4c, the 
skin color strength extraction step S03 is executed, after which the skin color strength image 8, 
the mask image 7 and the edge mask image 20 are input and the synthesis step S04c is executed. 

At the synthesis step S04c, the input skin color strength image 8, the mask image 7 and 
the edge mask image 20 are synthesized. Specifically, the multiplication process is executed 
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using the skin color strength image 8 generated by the skin color strength extraction step S03, 
the mask image 7 generated by the mask process and the edge mask image 20 generated by the 
edge mask process. By the execution of the synthesis step S04c, the masked skin color strength 
image 9c is generated. 

In the skin color area extraction process executed by the skin color area extractor 4c, the 
synthesis step S04c is executed, after which the skin color area correction step SOS is executed 
using the masked skin color strength image 9c, and the skin color area image 10c is output. 
[Operation/effects] 

The image correction unit 1c uses the edge mask image 20 at the synthesis step S04c. 
In the edge mask image 20, the skin color strength is set to a low value or zero for the extracted 
edge and the surrounding area. As a result, the synthesis step S04c can acquire the masked skin 
color strength image 9c with the skin color strength set low or to zero for the edge and the 
surrounding area. The skin beautifying process is executed using this masked skin color strength 
image 9c, and therefore while maintaining the freshness of the edge and the surrounding areas, 
i.e. the eyes, eyebrows, mouth, etc., the gradation process can be executed for the other skin 
color portions. Especially, the skin beautifying process can be effectively executed for the face 
image having the lipstick or the eyebrows (such as thin eyebrows) of substantially the skin color. 

Fig. 14 is a diagram showing the difference between the skin color strength image 8 and 
the edge mask image 20. Fig. 14(a) shows an example of the skin color strength image 8, and 
Fig. 14(b) an example of the edge mask image 20. In this case, the color of the eyebrows of the 
left person in the original image 6 is near the skin color, and therefore, in the skin color strength 
image 8, the skin color strength of the eyebrows assumes a value near the skin color. Also, the 
color of the lips of the right person in the original image 6 is near the skin color, and therefore in 
the skin color strength image 8, the skin color strength of the lips assumes a value near the skin 
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color. In this situation, even the eyebrows of the left person and the lips of the right person 
would be gradated so that a skin beautification image 12 with blurred eyebrows and lips would 
be obtained. In the edge mask image 20, on the other hand, the edges of the eyebrows of the left 
person and the lips of the right person are extracted, and therefore the skin color strength of the 
eyebrows of the left person and the lips of the right person assume a value far from the skin 
color. The use of the edge mask image 20 can thus maintain the freshness of the eyebrows and 
the lips which are not gradated. 
[Modifications] 

In the edge mask image generation step S08, the enlarging process, the inversion 
process and the reduction process may be changed in the order of execution as required. In the 
case where the reduction process is executed before the inversion process, however, the area of a 
low or zero pixel value (skin color strength) is not expanded outward, but the area high in pixel 
value (skin color strength) or 255 ("1" for the skin color strength) is expanded outward. 

Instead of the edge mask processor 19 and the edge mask image generating step S08, 
the face element mask processor (corresponding to the face element extraction means) and the 
face element mask image generating process may be executed to generate a face element mask 
image in place of the edge mask image 20. In the face element mask image generating process, 
the edge is not extracted, but the elements (face elements) included in the face of the object. 
These face elements are extracted, for example, by template matching. In the face element mask 
image, the skin color strength of the extracted face elements or the pixels within a predetermined 
range from these face elements are set low or to zero. 
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